User-interactive automatic translation device and method for mobile device

ABSTRACT

A user-interactive automatic translation device for a mobile device, includes: a camera image controller for converting an image captured by a camera into a digital image; an image character recognition controller for user-interactively selecting a character string region to be translated from the digital image, performing a character recognition function on the selected character string region based on an optical character reader (OCR) function and character recognition information to generate a text string; and user-interactively correcting errors included in the text string. Further, the user-interactive automatic translation device includes a text transmission controller for transmitting the error-corrected text string; and an automatic translation controller for receiving the error-corrected text string from the text transmission controller, performing a morpheme analysis, tagging, a structure analysis, a structure conversion, and a vocabulary conversion on the text string to generate a translation of the text string based on a grammar of a target language.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present invention claims priority of Korean Patent Application No.10-2009-0086064, filed on Sep. 11, 2009, which is incorporated herein byreference.

FIELD OF THE INVENTION

The present invention relates to a user-interactive automatictranslation device and method for a mobile device; and more specificallyto a user-interactive automatic translation device and method for amobile device which is capable of achieving a high-quality automatedtranslation of a word or sentence included in an image captured by acamera included in the mobile device, e.g., cellular phone, PDA, PDP andthe like with an enhanced translation accuracy by interaction with auser.

BACKGROUND OF THE INVENTION

Performance of an automatic translation device has been graduallyimproved, however, translation results of the automatic translationdevice still contains lots of errors or faults. The automatictranslation device performing a rule-based or pattern-based translationparticularly shows unsatisfactory translation results with unnatural orungrammatical sentences.

Some of these errors can be solved by improving separate modulesincluded in a translation engine, however, because the separate modulesdo not consider a sentence as a whole, the errors are still likely tooccur. Therefore, a function of automatically correcting errorsoccurring in a final translation is required to upgrade a performance ofan automatic translation device.

Further, most of automatic translation devices are mainly used fordesktop computers or servers. These types of automatic translationdevices generally perform an automatic translation on alreadydigitalized text files, web documents, PDF files and the like.

However, there exist various types of offline texts required to betranslated, e.g., menus for restaurants, sign boards on the street, hardcopy documents and the like.

Conventionally, there has been an automatic translation device for amobile device, which includes a character recognition module to providean automatic translation function.

However, the conventional automatic translation device has a shortcomingof a poor quality of translation due to a limitation of characterrecognition technologies.

SUMMARY OF THE INVENTION

In view of the above, the present invention provides a user-interactiveautomatic translation device and method for a mobile device having acamera. In accordance with the present invention, a character stringregion to be automatically translated can be selected by a user usingthe user interface in a still image captured by a mobile device having acamera. Then, the selected character string region is subjected to acharacter recognition process to be converted into a digital textstring. Accordingly, errors occurring in the character recognitionprocess can be directly corrected by the user feedback to generate anerror-corrected text string, which is in turn automatically translated.

In accordance with a first aspect of the present invention, there isprovided a user-interactive automatic translation device for a mobiledevice, including: a camera image controller for converting an imagecaptured by a camera into a digital image; an image characterrecognition controller for user-interactively selecting a characterstring region to be translated from the digital image, performing acharacter recognition function on the selected character string regionbased on an optical character reader (OCR) function and characterrecognition information stored in a character recognition information DBto generate a text string, and user-interactively correcting errorsincluded in the text string; a text transmission controller fortransmitting the error-corrected text string; and an automatictranslation controller for receiving the error-corrected text stringfrom the text transmission controller, performing a morpheme analysis,tagging, a structure analysis, a structure conversion, and a vocabularyconversion on the text string to generate a translation of the textstring based on a grammar of a target language, wherein the morphemeanalysis, the tagging, the structure analysis, the structure conversion,the vocabulary conversion, and the translation generation are performedbased on information for translation stored in an automatic translationinformation DB.

In accordance with a second aspect of the present invention, there isprovided a user-interactive automatic translation method for a mobiledevice, including: capturing an image of an object using a camera;converting the image of the object to a digital image;user-interactively selecting a character string region to be translatedfrom the digital image; performing a character recognition on theselected character string region based on an optical character reader(OCR) and character recognition information stored in a characterrecognition information DB to generate a text string; user-interactivelycorrecting errors included in the text string; and performing a morphemeanalysis, tagging, a structure analysis, a structure conversion, and avocabulary conversion on the error-corrected text string to generate atranslation of the text string based on a grammar of a target language,wherein the morpheme analysis, the tagging, the structure analysis, thestructure conversion, the vocabulary conversion and the translationgeneration are performed based on information for translation stored inan automatic translation information DB.

In accordance with an embodiment of the present invention, a characterstring region to be automatically translated can be selected by a userusing the user interface in a still image captured by a mobile devicehaving a camera. Then, the selected character string region is subjectedto a character recognition process to be converted into a digital textstring. Errors occurring in the character recognition process can bedirectly corrected by the user feedback to generate an error-correctedtext string, which is in turn automatically translated. Accordingly, itis possible to solve a conventional problem that shows a low quality ofautomatic translation due to inaccurate character recognition.

Further, it is possible for the user to conveniently perform ahigh-quality automatic translation of translation objects such as menusof a restaurant, traffic signs, various foreign books, manuals offoreign brand products and the like using a portable mobile device atanywhere nationwide and worldwide. Furthermore, it is possible toachieve a high-quality automatic translation by interacting with theuser to minimize character recognition error.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects and features of the present invention will become apparentfrom the following description of an embodiment, given in conjunctionwith the accompanying drawing, in which:

FIG. 1 is a block diagram illustrating a user-interactive automatictranslation device for a mobile device in accordance with an embodimentof the present invention;

FIG. 2 is a detailed block diagram illustrating the image characterrecognition controller of FIG. 1;

FIG. 3 is a detailed block diagram illustrating the automatictranslation controller of FIG. 1;

FIG. 4 is a view illustrating a character recognition error displaywindow in accordance with the embodiment of the present invention; and

FIGS. 5A to 5C are flow charts sequentially illustrating auser-interactive automatic translation method for a mobile device inaccordance with the embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments of the present invention will be described indetail with reference to accompanying drawings which form a part hereof.

FIG. 1 is a block diagram illustrating a user-interactive automatictranslation device 100 for a mobile device in accordance with anembodiment of the present invention. The user-interactive automatictranslation device 100 includes a camera image controller 101, an imagecharacter recognition controller 103, a text transmission controller 105and an automatic translation controller 107.

The camera image controller 101 digitalizes the image of an object,captured by a camera S1, e.g., a camera embedded in the mobile device togenerate a digital image, e.g., a digital still image and supplies thedigital image to the image character recognition controller 103.

As shown in FIG. 2, the image character recognition controller 103includes a region selection user interface unit 1031, a characterrecognition unit 1033, a text converting unit 1035, a userinteraction-based recognition error correcting unit 1037, and acharacter recognition information database (DB) 1039.

The region selection user interface unit 1031 provides a user with auser interface by which the user can select a character string region tobe automatically translated from the digital image supplied from thecamera image controller 101. In response to the user selecting thecharacter string region through a rectangular region selection tool, theregion selection user interface unit 1031 provides the selectedcharacter string region to the character recognition unit 1033.

The character recognition unit 1033 performs a characteristicrecognition on the character string supplied from the region selectionuser interface unit 1031 based on a function of an optical characterreader (OCR) and information for character recognition that is stored inthe character recognition information DB 1039. Then, the characterrecognition unit 1033 supplies a resultant character string to the textconverting unit 1035.

The text converting unit 1035 converts the character string to astandard text character string based on the American Standard Code forInformation Interchange (ASCII), and supplied the standard text stringto the user interaction-based recognition error correcting unit 1037.

Here, the ASCII-based standard text string from the text converting unit1035 may be a standard text string that may be optimally recognized. Theuser interaction-based recognition error correcting unit 1037 displayrecognition candidates for each word included in the text string on theuser interface (UI) so that the user, by himself, can correct errorsthat may occur when recognizing the text string. The user can directlycorrect the errors included in the text string with various input tools,e.g., a digital pen, a key board on the mobile device and the like. Theuser interaction-based recognition error correcting unit 1037 receivesthe corrected text string from the user and supplies it to the texttransmission controller 105.

The character recognition information DB 1039 stores various types ofinformation preset for character recognition.

The text transmission controller 105 supplies the corrected text stringsupplied from the user interaction-based recognition error correctingunit 1037 to the automatic translation controller 107.

As shown in FIG. 3, the automatic translation controller 107 includes asentence recognition pre-process unit 1071, a source language morphemeanalyzing and tagging unit 1073, a source language structure analyzingunit 1075, a source language/target language converting unit 1077, atarget language generating unit 1079, and an automatic translationinformation DB 1081.

The sentence recognition pre-process unit 1071 recognizes the correctedtext string supplied from the text transmission controller 105 sentenceby sentence and supplies the sentence-by-sentence recognized text stringto the source language morpheme analyzing and tagging unit 1073. Thesentence recognition pre-process unit 1071 can separate sentences notprocessable by the OCR from the text string.

The source language morpheme analyzing and tagging unit 1073 analyzessource language morphemes of the text string supplied from the sentencerecognition pre-process unit 1071 based on a source language morphemeanalysis dictionary, a probability dictionary, and a context probabilitydata that are stored in the automatic translation information DB 1081,performs tagging on the result in various methodology such as trigram orhidden markov model (HMM) to restore each of vocabularies included inthe text string into original form thereof, applies optimum parts ofspeech appropriate for the context to the recovered text string toprovide the resultant text to the source language structure analyzingunit 1075.

The source language structure analyzing unit 1075 analyzes a grammaticalcorrelation and a modification structure of vocabularies included ineach sentence of the text string supplied from the source languagemorpheme analyzing and tagging unit 1073 based on a structure analysisrule and a translation pattern that are stored in the automatictranslation information DB 1081 to provide the resultant text string tothe source language/target language converting unit 1077.

The source language/target language converting unit 1077 performs asource language-to-target language conversion on the text stringsupplied from the source language structure analyzing unit 1075 based onthe translation dictionary, the translation pattern, and a translationmemory that are stored in the automatic translation information DB 1081.Further, the source language/target language converting unit 1077converts vocabularies of source language to vocabularies of targetlanguage, and supplies the structure-converted and vocabulary-convertedtext string to the target language generating unit 1079.

The target language generating unit 1079 performs a process, such as aninflection process, on the text string supplied from the sourcelanguage/target language converting unit 1077 to be appropriate for agrammar for the target language to generate a final translation.

The automatic translation information DB 1081 stores the source languagemorpheme analysis dictionary, the probability dictionary, the contextprobability data, the structure analysis rule, the translation pattern,the translation dictionary, the translation pattern, the translationmemory and the like.

In accordance with an embodiment of the present invention, a characterstring region to be automatically translated can be selected by a userusing the user interface in a still image captured by a mobile devicehaving a camera. The selected character string region is subjected to acharacter recognition process to be converted into a digital textstring. Then, errors occurring in the character recognition process canbe directly corrected by the user feedback to generate anerror-corrected text string, which is in turn automatically translated.Accordingly, it is possible to solve a conventional problem that shows alow quality of automatic translation due to inaccurate characterrecognition.

A user-interactive automatic translation method performed in a mobiledevice configured as above will now be described in accordance with theembodiment of the present invention.

FIGS. 5A to 5C are flow charts sequentially illustrating auser-interactive automatic translation method performed in the mobiledevice in accordance with the embodiment of the present invention.

First, an image of an object is captured by the camera S1, e.g., aninternal camera included in the mobile device included in theuser-interactive automatic translation device 100 (S501).

Then, the camera image controller 101 receives the image of the objectfrom the camera S1 in step S503, generates a digital image in step S505,and supplies the generated digital image, e.g., digital still image tothe region selection user interface unit 1031 included in the imagecharacter recognition controller 103 in step S507.

The region selection user interface unit 1031 provides a user with auser interface function in step S509 by which the user can select acharacter string region to be automatically translated from the digitalimage supplied from the camera image controller 101.

The user uses a rectangular selection tool to select the characterstring region subjected to an automatic translation process. The regionselection user interface unit 1031 receives the character string regionselected by the user in step S511 to provide it to the characterrecognition unit 1033 in step S513.

The character recognition unit 1033 performs a characteristicrecognition on the character string supplied from the region selectionuser interface unit 1031 based on an OCR function and information forcharacter recognition stored in the character recognition information DB1039 in step S515. Then, the character recognition unit 1033 providesthe character-recognized character string to the text converting unit1035 in step S517.

The text converting unit 1035 converts the character string to anASCII-based standard text character string in step S519 and supplied thestandard text string to the user interaction-based recognition errorcorrecting unit 1037 in step S521.

The text converting unit 1035 may perform a text-conversion process onnot only optimum word recognition candidates (i.e., 401 in FIG. 4) inthe character string recognized by the character recognition unit 1033but also recognition candidates (i.e., 402 in FIG. 4) of each of wordsincluded in each optimum word recognition candidate.

Here, the ASCII-based standard text string from the text converting unit1035 may be a standard text string that may be optimally recognized. Theuser interaction-based recognition error correcting unit 1037 displayrecognition candidates for each word included in the text string on theUI so that the user, by himself, can correct errors that may occur whenrecognizing the text string in step S523.

The user may directly correct the errors included in the text stringwith various input tools, e.g., a digital pen, a key board on the mobiledevice and the like. The user interaction-based recognition errorcorrecting unit 1037 receives the corrected text string from the user instep S525 and provides the corrected text string to the texttransmission controller 105 in step S527.

For example, a process of correcting the errors can be performed by theuser on a character recognition error display screen as shown in FIG. 4.

More specifically, as shown in FIG. 4, the optimum word recognitioncandidates 401 for the ASCII-based standard text string converted by thetext converting unit 1035 are displayed on the character recognitionerror display screen. Further, the recognition candidates 402 for eachoptimum word recognition candidate 401 are also displayed downward inthe order from a high weight value to a low weight value on thecharacter recognition error display screen, wherein each recognitioncandidate 402 is enclosed in a rectangle. At this time, in the casewhere, among the optimum word recognition candidates 401, e.g., “word 1”is under a recognition error, the user can touch or click on, e.g.,“candidate 1-2”, one of the recognition candidates for word 1, tocorrect the error of character recognition.

The character recognition error display screen provides an input unit403 (e.g., key pad) and an input window 404 as shown in FIG. 4.Accordingly, in cases where an erroneous word does not belong to therecognition candidates appearing in the character recognition errordisplay screen, the user can enter a word with the input unit 403 whilerecognizing the entered word through the input window 404, therebycorrecting the character recognition error.

The text transmission controller 105 provides the corrected text stringsupplied from the user interaction-based recognition error correctingunit 1037 to the sentence recognition pre-process unit 1071 included inthe automatic translation controller 107 in step S529.

The sentence recognition pre-process unit 1071 recognizes the correctedtext string provided from the text transmission controller 105 in stepS531 sentence by sentence and provides the sentence-by-sentencerecognized text string to the source language morpheme analyzing andtagging unit 1073 in step S533.

The source language morpheme analyzing and tagging unit 1073 analyzessource language morphemes of the text string provided from the sentencerecognition pre-process unit 1071 based on a source language morphemeanalysis dictionary, a probability dictionary, a context probabilitydata and the like that are stored in the automatic translationinformation DB 1081, performs tagging on the result in variousmethodology such as trigram, hidden markov model (HMM) or the like torestore each of vocabularies included in the text string into originalform thereof in step S535, applies optimum parts of speech appropriatefor the context to the resultant text string, and provides the resultanttext to the source language structure analyzing unit 1075 in step S537.

The source language structure analyzing unit 1075 analyzes a grammaticalcorrelation and a modification structure of vocabularies included ineach sentence of the text string provided from the source languagemorpheme analyzing and tagging unit 1073 based on a structure analysisrule and a translation pattern that are stored in the automatictranslation information DB 1081 in step S539, and supplies the resultanttext string to the source language/target language converting unit 1077in step S541.

The source language/target language converting unit 1077 performs asource language-to-target language conversion on the text stringprovided from the source language structure analyzing unit 1075 based onthe translation dictionary, the translation pattern, a translationmemory and the like that are stored in the automatic translationinformation DB 1081. Further, the source language/target languageconverting unit 1077 converts vocabularies of source language tovocabularies of target language in step S543, and provides the resultanttext string to the target language generating unit 1079 in step S545.

The target language generating unit 1079 performs a process, such as aninflection process, on the text string supplied from the sourcelanguage/target language converting unit 1077 to be appropriate for agrammar for the target language to generate a final translation in stepS547.

The user-interactive automatic translation method in accordance with theembodiments of the present invention can be implemented ascomputer-executable codes or programs that are stored in acomputer-readable storage medium.

The computer-readable storage medium includes all types of storagedevices that store data readable by a computer system.

An example of the computer-readable storage medium includes ROMs, RAMs,CD-ROMs, magnetic tapes, floppy disks, optical data storage and thelike. Further, the computer-executable codes or programs can betransmitted over a network including the Internet, and distributablyexecuted in a plurality of computer systems interconnected over anetwork.

While the invention has been shown and described with respect to theembodiments, it will be understood by those skilled in the art thatvarious changes and modification may be made without departing from thescope of the invention as defined in the following claims.

What is claimed is:
 1. A user-interactive automatic translation devicefor a mobile device, comprising: a camera image controller forconverting an image captured by a camera into a digital image; an imagecharacter recognition controller for user-interactively selecting acharacter string region to be translated from the digital image,performing a character recognition function on the selected characterstring region based on an optical character reader (OCR) function andcharacter recognition information stored in a character recognitioninformation database (DB) to generate a text string, anduser-interactively correcting errors included in the text string; a texttransmission controller for transmitting the error-corrected textstring; and an automatic translation controller for receiving theerror-corrected text string from the text transmission controller,performing a morpheme analysis, tagging, a structure analysis, astructure conversion, and a vocabulary conversion on the text string togenerate a translation of the text string based on a grammar of a targetlanguage, wherein the morpheme analysis, the tagging, the structureanalysis, the structure conversion, the vocabulary conversion, and thetranslation generation are performed based on information fortranslation stored in an automatic translation information DB; whereinthe image character recognition controller includes: a region selectionuser interface unit for providing a user with an interface so that theuser selects the character string region from the digital image, andreceiving the character string region selected by the user; a characterrecognition unit for performing a character recognition function on thereceived character string region based on the OCR function and thecharacter recognition information stored in the character recognitioninformation DB; a text converting unit for converting thecharacter-recognized character string into a standard text string; and auser interaction-based recognition error correcting unit for displayingthe standard text string on a display window and to receive anerror-corrected text string through the user interface, and wherein theuser interaction-based recognition error correcting unit displaysoptimum word recognition candidates for the standard text string andrecognition candidates for each optimum word recognition candidate suchthat the recognition candidates are displayed downward under the optimumword recognition candidates in the order from a high weight value to alow weight value, each recognition candidate being enclosed in arectangle, and when one of the recognition candidates associated with anerroneous optimum word recognition candidate is selected through theuser interface, the erroneous optimum word recognition candidate isreplaced by the selected recognition candidate to correct the error ofcharacter recognition.
 2. The user-interactive automatic translationdevice for a mobile device of claim 1, when no recognition candidatesassociated with the erroneous optimum word recognition candidate exist,the user interaction-based recognition error correcting unit replacesthe erroneous optimum word recognition candidate with a word orcharacter entered by a user to correct the error of characterrecognition.
 3. The user-interactive automatic translation device for amobile device of claim 1, wherein the text converting unit converts theoptimum word recognition candidates for the character string and therecognition candidates for the optimum word recognition candidates intoa standard text string.
 4. The user-interactive automatic translationdevice for a mobile device of claim 1, wherein the automatic translationcontroller includes: a sentence recognition pre-process for recognizingthe text string sentence by sentence; a source language morphemeanalyzing and tagging unit for analyzing source language morphemes ofthe sentence-by-sentence recognized text string based on first automatictranslation information stored in the automatic translation informationDB, performing tagging on the result and restoring each of vocabulariesincluded in the text string into original form thereof to apply parts ofspeech to the recovered text string; a source language structureanalyzing unit for analyzing a grammatical correlation and amodification structure of vocabularies included in each sentence of therecovered text string based on second automatic translation informationstored in the automatic translation information DB; a sourcelanguage/target language converting unit for performing a sourcelanguage-to-target language conversion on the analyzed text string basedon third automatic translation information stored in the automatictranslation information DB to convert vocabularies of source language tovocabularies of target language; and a target language generating unitfor applying a grammar of target language to the structure-converted andvocabulary-converted text string to generate a translation.
 5. Theuser-interactive automatic translation device for a mobile device ofclaim 4, wherein the sentence recognition pre-process unit performs asentence separation function on the transmitted text string based on theOCR function.
 6. The user-interactive automatic translation device for amobile device of claim 4, wherein the first automatic translationinformation includes a source language morpheme analysis dictionary, aprobability dictionary and a context probability data.
 7. Theuser-interactive automatic translation device for a mobile device ofclaim 4, wherein the second automatic translation information includes astructure analysis rule and a translation pattern.
 8. Theuser-interactive automatic translation device for a mobile device ofclaim 4, wherein the third automatic translation information includes atranslation dictionary, a translation pattern and a translation memory.9. A user-interactive automatic translation method for a mobile device,comprising: capturing an image of an object using a camera; convertingthe image of the object to a digital image; user-interactively selectinga character string region to be translated from the digital image;performing a character recognition on the selected character stringregion based on an optical character reader (OCR) and characterrecognition information stored in a character recognition information DBto generate a text string; user-interactively correcting errors includedin the text string; and performing a morpheme analysis, tagging, astructure analysis, a structure conversion, and a vocabulary conversionon the error-corrected text string to generate a translation of the textstring based on a grammar of a target language, wherein the morphemeanalysis, the tagging, the structure analysis, the structure conversion,the vocabulary conversion and the translation generation are performedbased on information for translation stored in an automatic translationinformation DB, wherein user-interactively correcting errors includes:providing a user with a user interface so that the user selects thecharacter string region from the digital image; performing a characterrecognition on the selected character string region based on an OCRfunction and the character recognition information stored in thecharacter recognition information DB; converting thecharacter-recognized character string into a standard text string;displaying the standard text string on a display window through the userinterface; and correcting the text string through the user interface,and wherein displaying the standard text string includes displayingrecognition candidates for an erroneous optimum word recognitioncandidate to correct the erroneous optimum word recognition candidatesuch that the recognition candidates are displayed downward under theoptimum word recognition candidate in the order from a high weight valueto a low weight value, each recognition candidate being enclosed in arectangle.
 10. The user-interactive automatic translation method for amobile device of claim 9, wherein said correcting includes, when one ofthe recognition candidates associated with the erroneous optimum wordrecognition candidate is clicked through the user interface, replacingthe erroneous optimum word recognition candidate by the clickedrecognition candidate to correct the error of character recognition. 11.The user-interactive automatic translation method for a mobile device ofclaim 10, wherein said correcting includes, when no recognitioncandidates associated with the erroneous optimum word recognitioncandidate exist, replacing the erroneous optimum word recognitioncandidate with a word or character entered by a user to correct theerror of character recognition.
 12. The user-interactive automatictranslation method for a mobile device of claim 9, wherein saidconverting the character-recognized character string includes convertingthe optimum word recognition candidates for the character string and therecognition candidates for the optimum word recognition candidates intoa standard text string.
 13. The user-interactive automatic translationmethod for a mobile device of claim 9, wherein said generating thetranslation includes: recognizing the error-corrected text stringsentence by sentence; analyzing and tagging source language morphemes ofthe sentence-by-sentence recognized text string based on first automatictranslation information stored in the automatic translation informationDB to restore each of vocabularies included in the text string intooriginal form thereof; analyzing a grammatical correlation and amodification structure of vocabularies included in each sentence of therecovered text string based on second automatic translation informationstored in the automatic translation information DB; performing a sourcelanguage-to-target language conversion on the analyzed text string basedon third automatic translation information stored in the automatictranslation information DB to convert vocabularies of source language tovocabularies of target language; and applying a grammar of targetlanguage to the structure-converted and vocabulary-converted text stringto generate the translation.
 14. The user-interactive automatictranslation method for a mobile device of claim 13, wherein the firstautomatic translation information includes a source language morphemeanalysis dictionary, a probability dictionary and a context probabilitydata.
 15. The user-interactive automatic translation method for a mobiledevice of claim 13, wherein the second automatic translation informationincludes a structure analysis rule and a translation pattern.
 16. Theuser-interactive automatic translation method for a mobile device ofclaim 13, wherein the third automatic translation information includes atranslation dictionary, a translation pattern and a translation memory.